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Abstract 

Background: Computer-based examinations (CBE) ensure higher efficiency with respect to producibility and 
assessment compared to paper-based examinations (PBE). However, students often have objections against CBE 
and are afraid of getting poorer results in a CBE. 

The aims of this study were (1) to assess the readiness and the objections of students to a CBE vs. PBE (2) to 
examine the acceptance and satisfaction with the CBE on a voluntary basis, and (3) to compare the results of the 
examinations, which were conducted in different formats. 

Methods: Fifth year medical students were introduced to an examination-player and were free to choose their 
format for the test. The reason behind the choice of the format as well as the satisfaction with the choice was 
evaluated after the test with a guestionnaire. Additionally, the expected and achieved examination results were 
measured. 

Results: Out of 98 students, 36 voluntarily chose a CBE (37%), 62 students chose a PBE (63%). Both groups did not 
differ concerning sex, computer-experience, their achieved examination results of the test, and their satisfaction 
with the chosen format. Reasons for the students' objections against CBE include the possibility for outlines or 
written notices, a better overview, additional noise from the keyboard or missing habits normally present in a 
paper based exam. The students with the CBE tended to judge their examination to be more clear and 
understandable. Moreover, they saw their results to be independent of the format. 

Conclusions: Voluntary computer-based examinations lead to egual test scores compared to a paper-based 
format. 
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Background 

The use of computer-based examinations combines 
advantages with respect to content (integration of other 
media, favourable presentation of pictures, and possibi- 
lity of other examination formats) with rapid data analy- 
sis. This promises higher efficiency with respect to 
implementation and evaluation [1-3]. However, students 
often have worries and prejudices concerning an unsa- 
tisfactory graphical user interface (GUI) of the examina- 
tion software, possible technical problems with the 
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computer, concentration problems, and additional exam 
stress [4]. An increase of the number of required graded 
examinations during medical studies from zero to 39 
has been one effect of the new medical licensing regula- 
tions in Germany [5]. This presents a challenge to aca- 
demic departments, especially those with limited 
teaching personnel and financial resources. To our 
knowledge this is the first study comparing voluntary 
computer-based examinations (CBE) and paper-based 
examinations (PBE). The aims of this study were (1) to 
assess the readiness of students to a CBE vs. PBE (2) to 
examine the acceptance and satisfaction with the CBE 
on a voluntary basis, and (3) to compare the results of 
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the examinations, which were conducted in different 
formats. 

Methods 

The examination in human genetics in the University of 
Heidelberg is usually performed paper-based. For our 
study 98 fifth-year medical students were offered a 
choice between CBE and PBE for a written human 
genetics exam after a two week course. All of the stu- 
dents were introduced to both options at the beginning 
of the course. Due to concerns of a possible disadvan- 
tage in a CBE, all students received a short introduction 
to the examination software (Campus player version 
181205 [6]). Suggestions for improvement concerning 
possible disparities of the GUI compared to the PBE 
were implemented in a new update. Afterwards, the stu- 
dents were free to decide for an examination format. 
The test consisted of 26 questions, 24 multiple-choice- 
format with differing types (type A and Kprim) and two 
short answer questions. The test lasted 90 minutes in 
both examination formats. Both formats consisted of 
identical questions which could be answered in optional 
order. After the test, the students answered a question- 
naire with 19 items to evaluate their acceptance and the 



reason for their choice as well as the usability of the 
software on a Likert scale from 1 to 5 (see table 1). 
Furthermore, questions concerning previous experience 
with CBE and opinions thereof were asked. 

For statistical evaluation a t-test with Satterthwaite- 
correction for unequal variances was conducted. The 
statistical evaluation was performed with SAS 9.1. 

Results 

36 (37%) of the students chose the computer-based 
examination (CBE) and 62 (63%) chose the paper-based 
examination (PBE). The ratio of male to female students 
in both exam formats was similar (CBE: m = 14, f = 22; 
PBE: m = 26, f = 36; not statistically significant). On 
average, both groups had the same level of previous 
experience with CBE (CBP: 3.7; PBE: 3.5; not statistically 
significant), which is very little compared to the numer- 
ous PBEs in the curriculum. In the event of a repeat 
exam, the overwhelming number of the students who 
took CBE would choose the CBE again (3.7 on a Likert 
scale), only a few students (6 out of 36, ca. 15%) stated 
a preference for PBE in future exams. The PBE-group 
showed a very high disposition to maintain the PBE-for- 
mat (4.6). None of the students in this group stated a 



Table 1 Questionnaire 

Mean StdDev Mean StdDev Pr > |t| 

computer-based paper-based 

examination examination 

format (n = 36) format (n = 62) 



1) The usability of the examination was easy. 


4.556 


0.843 


4.667 


0.636 


0.5004 


2) The examination was clear and easily understandable. 


4.361 


1.073 


4.000 


1.118 


0.1240 


3) Additional mental effort was required due to the chosen examination format. 


2.306 


1.390 


1.123 


0.381 


<.0001 


4) I found it useful to take the examination in the chosen format, because it increased efficacy in this 


2.972 


0.941 


4.375 


0.822 


<.0001 


situation. 












5) I was anxious before the examination. 


3.556 


1.054 


3.789 


1.048 


0.2995 


6) After a few questions, my anxiety at the beginning of the examination was gone. 


3.528 


1.055 


3.211 


1.264 


0.1949 


7) If I had a choice, I would take future examinations more often in the chosen format. 


3.722 


1.111 


4.596 


0.593 


<.0001 


8) Expected scoring in examination 


18.72 


2.387 


17.15 


3.664 


0.0218 


9) CBE: If I would have chosen PBE I suppose my results would have been (better/equal/worse). PBE: 


0.086 


0.284 


-0.452 


0.563 


<.0001 


If I would have chosen CBE I suppose my results would have been (better/equal/worse). 












10) In how many CBEs did you participate so far? 


2.250 


1.180 


2.500 


1.059 


0.3016 


1 1) It is an advantage of CBE that there is the possibility to change my answers during the 


4.371 


0.770 


3.344 


1.263 


<.0001 


examination. 












12) It is important for me to have the possibility to change my answers during the examination. 


4.200 


0.933 


2.900 


1.231 


<.0001 


13) 1 found it useful to take this examination in CBE-format because it prepared me for further 


3.657 


1.259 








upcoming CBEs. (only CBE) 












14) The CBE itself was in total better than I expected it to be. (only CBE) 


3.417 


1.025 








15) There were no technical problems in the CBE. (only CBE) 


4.722 


0.615 








16) I was satisfied with the graphical user interface, (only CBE) 


4.714 


0.667 








1 7) Reason for preference (open question) 












18) suggestions for improvement (open question) 












19) I decline CBE in general, (only PBE) 






2.491 


1.269 





items 1-7 and 11-17: Likert scale with scoring from 'I completely agree' (5) to 'I do not agree at all' (1), items 8 and 10: numerical scale, item 9: +1 (better), 0 
(equal), -1 (worse), items 18 and 19: open question. For statistical evaluation a t-test with Satterthwaite-correction for unequal variances was conducted. 
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preference for the CBE format in future tests. In addi- 
tion, the reason for the choice of the format was evalu- 
ated in an open-ended question. The results are 
presented in table 2. 

According to the evaluation of both groups, predomi- 
nantly no or only very little additional effort was neces- 
sary to handle the format, whereas the students of the 
CBE-group noted higher mental exertion due to the 
CBE format (CBE: 2.3, PBE: 1.1, p < 0.001). In the CBE- 
group, the members felt no influence over their perfor- 
mance in the examination, whereas members of the 
PBE-group perceived an added benefit with respect to 
their performance in the examination (CBE: 3.0, PBE 
4.4; p < 0.001). 

Dealing with the examination format was judged to be 
simple in both groups (CBE: 4.6 vs. PBE: 4.7, not statis- 
tically significant), although the CBE was supposed to be 
more clear and easier to understand (CBE: 4.4 vs. PBE: 
4.0, p = 0.12). There were no technical problems in the 
CBE-group (4.7). In addition, the students in the CBE 
were very satisfied with the graphical user interface 
(4.7). 

There was no difference between the two formats con- 
cerning anxiety before (CBE: 3.6 vs. PBE: 3.8, n.s.) as well 
as the reduction of anxiety after the examination (CBE: 
3.5 vs. PBE: 3.2, n.s.). While the students in the CBE- 
group judged their results in both formats equally, stu- 
dents in the PBE-group had more anxiety of getting 
poorer results in the computer-based examination. The 
expected score directly after the examination was higher 
in the CBE-group, however only 1.5 points in average 
(total 26 points, CBE: 18.7 points vs. 17.2 points, p < 
0.03, see figure 1). The results in both groups were equal, 
which could be demonstrated in the average number of 
points (CBE: 18.9 vs. PBE: 18.5, n.s.), discrimination of 
grades (see figure 2) and the number of students who 
failed the examination (CBE: 2 vs. PBE: 8, n.s., minimum 
15 points). 

From the instructors' point of view, the assessment 
of the computer-based examination was quicker and 
more efficient (0.5 hours for one instructor for 36 stu- 
dents vs. 60 hours for 62 students). The assessment 
took 45 min at the CBE for the two open ended 
questions. 

Table 2 Reason for preference 

Reason for the preference (frequency of the item) of the computer- 
based examination format n = 26 of 36 

Quick assessment (n = 1 0, 34%) 

Supporting technological advancement (n = 7, 27%) 

Support of the computer concerning the number of the items to consider 
(n = 7, 27%) 

More objective assessment (n = 2, 8%) 



Discussion 

Primary results show a high acceptance of the compu- 
ter-based examination, which is reflected in a 37% 
voluntary participation and in a high level of readiness 
to take further examinations in this examination for- 
mat. The students of the paper-based examination 
showed very high willingness to remain in this exami- 
nation format for upcoming examinations. Addition- 
ally, this group showed a high level of anxiety with 
respect to poorer performance in a computer-based 
format. On the one hand, this could be a consequence 
of being used to paper-based examinations. On the 
other hand, it could be due to additionally stated rea- 
sons, which should be taken seriously and diminished 
in future examinations as effectively as possible (see 
table 2). 

For example, there should be the possibility for writ- 
ten notices on the test (possibility for outlines, personal 
remarks etc.), including the computer-based examina- 
tion. This was a motivating reason for 23 students who 
registered for the paper-based examination. On average, 
the anxiety before the examination was comparable in 
both groups. The fear of PC-errors or technical difficul- 
ties was nevertheless reason enough for the choice of 
seven students for the paper-based examination. This 
anxiety could possibly be lowered by an additional intro- 
duction to the examination software or a sample exami- 
nation for these students. The perceived usefulness of 
the paper-based examination concerning a personal per- 
formance increase in this format could surely be inter- 
preted in the context with the given reasons for the 
decision to take the paper-based examination. One rea- 
son may be the possibility of writing short notices on 
the test form or a subjectively better overview of the 
examination can lead to the impression of a higher per- 
formance. Moreover, Miller et al. found that the devel- 
opment of visually rich quizzes was greatly facilitated by 
the use of computers [7]. While Ogilvie et al. demon- 
strated that students found computer based tests less 
time consuming [8], we experienced that both groups 
finished in nearly the same amount of time. 

The slightly higher mental exertion required in the 
computer-based format could be explained by the stu- 
dents' familiarity with paper-based examinations and the 



Reason for the preference (frequency of the item) of the paper- 
based examination format n = 55 of 62 

Possibility for outlines or written notices on the questionnaire 
(n = 23, 42%) 

Better overview (n = 1 5, 27%) 

Habit (n = 11, 20%) 

Fear of computer errors or disturbingly loud keyboards (n = 6, 1 1%) 
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Figure 1 Expected and achieved scoring of the exam takers 




Achieved scoring 



rare usage of the graphical user interface. Regular com- 
puter-based examinations in the curriculum of the 
faculty in more subjects with the same examination soft- 
ware could possibly change this impression of the stu- 
dents by familiarizing these students with computer- 
based examinations and lower the anxiety. On the other 
hand, the objection of some students concerning dis- 
turbingly loud keyboards must be taken seriously (see 
table 2) and be corrected with the use of special key- 
boards if the occasion arises. 



The examination was judged as simple, there were no 
technical difficulties, and the students were satisfied 
with the graphical user interface. The usability was 
judged to be very high in the CBE-group, especially in 
clarity and understanding. The students with the CBE 
judged their examination to be more clear and under- 
standable, however this difference did not reach the 
level of significance. This is evidence of a high level of 
security throughout the examination, which is in con- 
cordance with other studies [9-12]. Due to the fact that 




grades 1 2 3 4 failed 



I | Computer-based form at 
[~1 Paper-based format 

Figure 2 Distribution of grades in computer-based and paper-based examination 
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answers to open ended questions could be corrected by 
only one instructor, the quality criteria of equal treat- 
ment could be better achieved and as a consequence 
objectivity could be raised. 

Another essential result of this study is the indepen- 
dence of the exam outcome from the chosen format. 
In addition, the average score depicted as the distribu- 
tion of the grades showed no difference. Russell & 
Haney show that the test results of students accus- 
tomed to writing on computer are higher then those 
written by hand [13]. Despite this, there was a ten- 
dency of more students to fail the examination in the 
PBE-group (8 vs. 2 in the CBE-group, 12.9% vs. 5.6%). 
However, this is not statistically significant. On the 
one hand we could not principally rule out a potential 
bias that the more intellectual students rather chose 
the CBE, so possible disadvantages due to technical 
reasons were compensated. On the other hand the bet- 
ter self-assessment of the students in the CBE-group is 
impressive and may hint towards a more optimistic 
attitude of these students to support innovation from 
the beginning. 

The main advantage of the computer-based exami- 
nation is an increase in efficiency and objectivity, 
because the automatic procession of the examination 
data is assumed to be less error-prone. Peterson et al. 
pointed out that an important step in evaluating com- 
puter-based examinations is to be sure that the exam 
format is measuring the examinee's knowledge and 
not their comfort level or confidence with the tech- 
nology [14]. Even script concordance tests could be 
examined computer-based [15]. Some studies showed 
that a development of a web-based assessment 
resulted in less administration for course organizers 
[11,16]. Unfortunately our study design did not allow 
performing a randomised trial because of legal reasons 
so we focused on the voluntary aspect. Here we found 
no differences if the students are free to choose the 
test format. Since this study was not intended to 
prove that CBE is equal or superior to PBE for all 
exam takers, further studies need to test how these 
results and the evaluation of a computer-based exami- 
nation (totally or with randomised access) influence 
the acceptance, the assessment of the usability, and 
the outcome of the examination, especially in those 
students who do not prefer a computer-based 
examination. 

With respect to the variety of teaching and examina- 
tion content, a computer-based examination is not only 
equally in its efficiency and ability to measure academic 
performance, but also an instrument to examine applied 
knowledge and visual skills with the help of innovative 
questionnaires and the use of complex media and/or 
new item structures such as the key feature. 



Conclusion 

Despite the exam anxiety on the part of the students, 
37% chose a computer-based examination format. In 
total, there were only very few students (ca. 5%) who 
denied the computer-based option. 

We could show that voluntary computer-based exami- 
nations lead to equal test scores compared to the paper- 
based manner. After further improvement and compen- 
sation of objectives on the side of the students, a 
required computer-based format should be of no disad- 
vantage to the students. By providing reliable informa- 
tion and a proper preparation of the students for the 
exam via an introduction to the software, a CBE could 
be a good method to conduct written examinations effi- 
ciently and fairly. 
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